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The Evaluation Productivity Project, scheduled to run 
through November 1985, was initially intended to explore the 
relationship between costs and the extent of utilization of 
evaluation. The original plan called for the development of a 
set of case scenarios during FY 1984, to be used in the empirical 
research of FY 1985. Each of these scenarios was to be "coated" 
in terms of its direct and indirect costs. Subsequently, the 
plan was altered to give greater emphasis in FY 1985 to 
additional synthesis and dissemination of the project's 
longstanding research on evaluation utilization. This change in 
emphasis, along with the decision not to undertake new empirical 
research, rendered the planned costing exercise pointless.. 
Nonetheless, it seems appropriate to set 'down here some of the 
reflections on evaluation costs that emerged during the initial 
stages of work. (See also Alkin & Solmon, The Costs of 
Evaluation , 1983.) 

This paper, then, summarizes our views on the costs of 
evaluation and, in particular, our hunches about the kinds of 
costs associated with factors known to affect utilization. The 
first section deals with general issues involved in identifying 
and valuing cost components. We will then discuss the 
appropriate costs of evaluation and consider costs in relation to 
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benefits. The final section spells out the cost implications of 

I 

high-utilization factors. 

Identifying and Valuing E valuation Costs 

Before we consider how the costs o^ evaluation can be 
identified and valued^ a distinction must be made between direct 
and indirect costs. "Direct costs" are simply the direct cash 
outlays ■ necessary for initiating and implementing the 
evaluation. Examples include salaries » travel expenses » and the 
cost of test booklets and data processing. Because direct costs 
are easy to identify, they are often regarded as synonymous with 
the costs of evaluation. 

Like other educational program activities, however, 
evaluations involve indirect as well as direct costs. While our 
discussion of costing focuses on the direct costs of evaluation, 
some consideration of the definition of indirect costs, and of 
general issues^ related to their identification and inclusion in 
the overall "cost" package, seems warranted. 

Most authorities (Alkin & Stecher, 1983; Catterall, 1983; 
Haggart, 1983; Morell & Weirich, 1983; Sander's, 1983; Solmon, 
1983) recognize the importance of these less obvious costs. But 
there is confusion in the literature over exactly how indirect 
costs should be defined. As a result of both this ambiguity and 
also their tendency to be less visible, indirect costs are often 
overlooked or underestimated when the costs of evaluation are 
computed . 



"Indirect costs'* can be defined in one of three ways. 
According to one point of view (Morell & Weirich, 1983), indirect 
costs are those dollar costs not specifically incurred by the 
evaluation project but shared across projects within the larger 
organizational structure: clerical time, facilities, 
communication, and even, in some instances, the salaries of 
project staff. This definition is probably the most 
straightforward. Indirect costs of this type are often 
calculated as a percentage of the direct costs of the evaluation 
and are included as a separate line item in the overall budget. 

According to a second point of view (Catterall, 1983), 
indirect costs are the opportunity costs of the evaluation: the 
use of equipment that could be used in other ways? volunteer time 
that could be spent on other projects? pupil time^that could be . 
spent in learning instead of testing? and the time the state 
agency spends in legislating, developing, and monitoring the 
evaluation. Almost every party involved in an evaluation could 
be engaging in other, possibly more valuable, activities. Thus, 
the opportunity cost is the best alternative use of a resource 
u«cd in the evaluation. Interpreted in this way, indirect costs 
can be included in the cost package as the dollar equivalent of 
-Iternative uses of evaluation resources, though Catterall 
suggests that these indirect costs are often better presented in 
their natural units (for instance, as the number of hours 
students could spend learning) rather than their dollar 
equivalents • 



Yet a third point of view (Solmon, 1983) holds that indirect 
costs are the side effects of the evaluation. For example, when 
a project concentrates on achieving immediately observable 
objectives in anticipation of an evaluation, certain long-term 
project goals may be slighted or even sacrificed altogether. 
Thus, the evaluation can be said to entail indirect costs. (It 
should be noted that Solmon considers opportunity costs 
separately from indirect costs.) 

As mentioned above, indirect costs in the first sense — 
shared costs of projects within an organization — are relatively 
easy to calculate, since they represent dollar costs for 
materials, personnel, and so forth. Calculating indirect costs 
in the second sense — opportunity costs — is somewhat more 
difficult, since one must determine alternative uses of resources 
and their corresponding values. It is indirect costs in the 
third sense as sic'e effects — that are most difficult to 
calculate; and we are not aware of systematic attempts to include 
such indirect costs as part of a total evaluation cost package. 

Several authors (Cattei^ll, 1983; Haggart, 1983; Levin, 
1983) have outlined the procedures necessary to identify the 
costs associated with an evaluation. According to Haggart, the 
process begins with a definition of the scope of the evaluation. 
The scope depends on the extent of the education intervention, 
the level of decision-making involved (the higher the level, the 
greater the scope of the evaluation), the purpose of the 
evaluation, and the complexity of the evaluation design. These 
aspects of the evaluation provide the context within which 




appropriate cost categories may be identified. 

Most authorities agree on the specific cost categories to be 
employed. For example, .the cost category systems presented by 
Alkin and Stecher (1993), Morell and Weiricb (1983), and Sanders 
(1983) are highly similar, differing only in detail. Drawing on 
the input of a large sample of professional evaluators, Alkin and 
Stecher (1983) delineate seven typical cost categories: 
professional staff, clerical staff, external consultants, 
materials/supplies/telephone, data processing, facilities, an^ 
travel. Similarly, Morell and Weirich (1983) break down 
evaluation costs into the following categories: personnel 
(evaluation staff, consultants, program staff, and subjects), 
technological capabilities (data processing, telephone services, 
and duplication) travel, office space and furnishings, supplies, 
and dissemination of/ findings (graphic artists, printers, and 
audiovisual specialists). The most detailed listing of cost 
categories is that provided by Sanders («^1983): evaluation staff 
salary and benefits, consultants, travel and per diem, 
communication, printing and duplication, data processing, printed 
materials, office supplies, subcontracts (outside of consulting), 
and overhead. Sanders expands on the types of costs frequently 
involved in data processing: systems design, data coding and 
checking, data storage and retrieval, computer programming, 
computer use for- manipulating or analyzing data, and 
computer-based bibliographic searches. 

After the cost categories have been identified, the next 
step is to determine values for each of these categories. One 



method for valuing "ingredients" is described by Levin (1983 
who offers the following hints about costing: 
Personnel 

Salaries and fringe benefits (percentage of 
.salaries): Value determined .by ' marketplace 
prices. When ■ personnel costs cannot easily be 
ascertained, use estimates of market value for 
similar services. 

Facilities 

Relited/leased facilities: Value determined by 
annual cost of expenditure. 

Owned facilities: Value determined by cost for 
similar space or annual cost (taking depreciation 
and interest on remaining undepreciated value into 
account ) . ' 

Equipment 

Purchased: Value determined in the same way as 
for facilities, depending on whether leased or 
owned. 

Donated/Borrowed: Value determined as cost of 
leasing or renting similar equipment. 

Supplies 

Value determined by adding estimated expenditures 
to estimated value of contributed supplies. 

^ Client Inputs 

Service: Value determined by total expense 
associated with service or by market • price of 
service provided. 



Note thsc these valuing procedures can be adapted to any o 
the cost-category frameworks described above. 



Using a variation of Levin's ingredients approach, Catterall 
(1983) offers an analysis of tho costs of testing that can be 
generalized to evaluation. The first step is to do a cost 
inventory. Again, the cost-category frameworks described above 
constitute alternative ways of compl3ting the cost n.ventory. 
The second step is to total the costs, which involves determining 
the actual monetary figure for each of the cost categories. 
(Levin's procedures outlined above provide a methodology for this 
step.) The last step in Catterall 's process is to locate the 
cost: that is, to decide who will pay for a particular component 
— the sponsor, a government agency, private party, the clients 
or subjects. An examination of Catterall 's variation of the 
Levin Ingredient Chart (Levin, 1975? see Figure 1) provides 
insights into the distributed features of evaluation costs. This 
chart is particularly informative with respect to indirect 
costs. Note, for example, that most of the cost-associated 
columns (particularly columns four, five, and six) will usually 
list indirect costs, although there may be some direct costs 
related to contributed private inputs. The last row in the 
chart, client time and other client inputs, presents a set of 
costs, ali^ of which are likely to be indirect. In addition, many 
facilities and equipment costs would be considered indirect if 
accounted for in the costs of conducting evaluations. 
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Source: Adapted from Levin » 1975; p. 101. 

Figure 1. Illustrative Framework for Cost Accounting in 
Educational Programs. 



Appropriate Costs of Evaluation 

Clearly, evaluations carry both direct and indirect costs, 
and both can be calculated. But iust what costs are ' 
"appropriate" (i.e.., what level of resources should be devoted to 
asking and answering questions about how well programs work or 
what they achieve)? This question can be approached in two 
ways. At a macro leval, ons> can compare the cost of evaluation 
with the cost of other program elements. That is, one can ask 
what the total costs of an evaluation should be relative to the 
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entity being evaluated. At a micro level, one can look at the 
way in which direct costs are allocated among the various budget 
categories and at the nature of the indirect costs associated 
with the conduct of the evaluation. Here we might ask what 
particular sorts of costs are worth incurring, given the nature 
of our evaluative interest. 

As noted in The Costs of Evaluation (Alkin & Solmon, 1983), 
the literature provides very few guidelines with respect to the 
macro level. Rusnell (1979) indicates that the evaluation cost 
should 5|iount to 10 percent of program cost, a figure that had 
been recommended during the early years of the federal Title VII 
pror-rams under the Elementary and Secondary Education Act 
(ESEA). More specifically, in the operation of these 
bilingual/bicultural programs during the early 1970s, it was 
suggested that funds for evaluation activities should constitute 
8 percent and funds for audit activities should constitute 2 
percent of programs costs. More recently, Drezek and his 
associates (1982) surveyed 55 LEAs and 14 SEAs and reported the 
proportions of^ funds allocated for evaluation in the various 
program proposals of those agencies. As expected, the 
percenta^ies varied substantially from one program to another? the 
median low was 1.5 percent and the median high was 5.5 percent. 
In addition, respondents to the survey recommended a 4-8 percent 
range as desireable. 

Guidelines at the micro level are even more scarce. In a 
study designed to gather data on the direct costs of the typical 
evaluation, Alkin and Stecher (1983) asked a nationwide group of 
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evaluators to conisider the costs' associated with two types of 

evaluation (process/implementation and outcorae/summative ) and 

.with three budget levels ($25,000, $10,000, and $4,000). Further 

constraints were imposed on the hypothetical examples, to insure 

.that the cost estimates would be comparable. Overall (that is, 

for both types of evaluation and at all three budget levels), the 

average shares recommended for each category were these; 

Professional staff 70% 
Clerical/secretarial staff 16% 
External consultants 2% 
Materials, supplies, telephone 5% 
Data process sing 3% 
Facilities 0% 
Travel 3% 

The specific recommended cost breakdowns varied widely, 
depending in part on evaluation type and on total budget level. 
Many of these differences are easily explained. For example, the 
percentage of direct costs allocated for travel was four to five 
times higher in implementation/process evaluations than in 
outcome/summative evaluations. Process evaluations typically 
require more site i^isits and more consultation with people in the 
field than do su;«native evaluations. 

Total budget level made a difference with respect to three 
of the seven cost categories. The first was data processing: 
"Data processing expenditures rose dramatically as the total 
evaluation budget increased from $4,000 to $10,000 and continued 
to rise in dollars (but not as a percentage of the total budget) 
when the total budget increased to $25,000" (Alkin & Stecher, 
1983, p. 7). One would expect data-processing costs to be low in 
a total evaluation budget of $4,000, since most of the analytic 
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work connected with such a low-cost evaluation would probably be 
done on a hand calculator. Moreover, the percentage of the 
budget allocated to data processing would probably not increase 
beyond a certain levels due to economies of scale: The initial 
outlay is high, but incremental costs for added units are 
relatively low. 

The secon^S^t category where the proportionate allocation 
increased with the size of the total budget was travels 0 
percent of the $4,000 budget, 2-3 percent of the $10,000 budget, 
and 5-8 percent of the $25,000 budget. Similarly, the proportion 
allocated for the third cost category — clerical and secretarial 
staff — rose as the size of the budget increased, from only 5 
percent at the $4,000 level to 20 percent at the $25,000 level. 
These systematic differences make clear the exten^*^o which 
indirect dollar costs are likely to be present in various 
evaluation budgets. The amount of data preparation, reporting, 
and just plain bureaucratic red tape seems to grow as the scope 
of the evaluation grows. Thus, a greater proportion of 
secretarial time is required at higher budget levels. Another 
potential explanation — and one that may be more relevant — is 
that, at small budget (or direct cost) levels, secretaries and 
clerical personnel are not easily divisible into arbitrary 
smaller budgeting units. Thus, these services are often provided 
by other projects or even by the school district itself. As a 
result, the secretarial costs of small evaluation projects tend 
to be disregarded when costs of evaluations are contemplated. 
They actually represent an indirect cost which may, on the basis 



o£ our findings, be fairly substantial in small'-scale 
evaluations. 

Our research on the costs of evaluation has convinced us 
that it is difficult, if not impossible, to- generalize aboo^ the 
direct or indirect costs associated with evaluations of different 
types. Idiosyncratic features affect the costs of evaluation and 
the extent of direct and indirect costs. These features can be 
grouped into four categories which, though occasionally 
overlapping, offer a convenient schema for purposes of 
discussion: context, administrative organization, support 
services and facilities, and administrative expectations and 
predispositions. 

One example that immediately comes to mind with respecf to 
context is choosing between an internal and an external 
evaluator. The selection of an internal evaluator may impose 
indirect cost burdens on other units within the organization. 
Conversely, if an external evaluator is chosen, many more of the 
costs will be direct because of the necessity for a contract. 

The administrative organization of the school district may 
impose different constraints upon the evaluation and thus affect 
its cost. If a complicated variety of approvals (for 
instruments, testing schedules, and so forth) and other 
administrative procedures are required, not only w'.ll the 
evaluator have to spend more time on the evaluation (a direct 
cost) but also other people in the organization will have to 



spend more of their time processing requests and making approvals 
(an indirect cost). 

Likewise, the nature of the support services and facilities 
available within a school district may have cost implications. 
For instance, the availability of computer services converts what 
would otherwise be a direct cost into an indirect cost. 
Similarly, if the evaluator has access to unused or underutilized 
facilities within the district, then the direct cost of renting 
office space will be saved. 

Finally, the expectations of progfam administrators — and 
especially their predisposition to- evaluation use — can affect 
both the direct and the indirect costs of an evaluatio.. 
Clearly, if the administrators who are the potential users of 
evaluation findings are hostile toward the evaluation and 
inclined to dismiss its findings, the evaluator will have to 
spend more time and energy trying to modify these 
predispositions, thus increasing both direct and indirect costs. 
(This issue is discussed further in the next section.) 

Costs in, Relation to Benefits 

Most authorities agree that the real issue is neither the 
absolute dollar cost of an evaluation nor the cost of the 
evaluation relative to the cost of the program being evaluated. 
The real issue is the relationship between the costs incurred by 
and the benefits that accrue from the evaluation. For example, 
Scriven ( 1974) notes that evaluations should be •'cost free," 
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implying that they should at least pay for themselves in the 
benefits they provide. Clearly, Scriven is advocating that costs 
and benefits should be compared by subtracting costs from 
benefits (B-C). Similarly, in Standards for Evaluations of. 
Educational Programs , Projects , and Materials (1981), the Joint 
Committee on Standards, for Educational Evaluation says that "the 
evaluation should produce information of sufficient value to 
justify the resources expended " (p. 60? emphasis added). The 
Joint Committee's guidelines further indicate that one should 
thoroughly investigate the costs and benefits of an evaluation 
before deciding to undertake it, adding that the evaluator should 
"conduct evaluations as economically as possible" (p. 61). 

If one accepts this point of view, a critical question 
becomes: How can the benefits deriving from an evaluation be 
determined? In other words, what constitutes ^an appropriate 
measure of benefits? Both Scriven and the Joint Committee would 
probably say that the dollar savings produced by the evaluation 
is the most appropriate measure. If the evaluation has resulted 
in recommendations as to how the program can be conducted in a 
less costly manner (with no reduction in the quantity or quality 
of educational outputs), and if the cost savings exceed the costs 
of the evaluation, then the cost of the evaluation is justified. 

This simple notion has been successfully practiced by Steven 
Frankel, Director of the Montgomery County (Maryland) Department 
of Education Accountability. In essence, by selecting for 
evaluation only those projects and services which are readily 
amenable to the demonstration of cost savings (primarily business 



service activities) ^ his Department has dramatically increased 
its total budget for evaluation and has freed up resources for 
conducting evaluations in which cost savings are not involved. 

The simple dollar-savings criterion is. obviously inadequate 
in those instances where cost economies cannot be demonstrated 
but where the evaluation still confers benefits, though of a less 
monetary nature. For example, an evaluation may result in 
recommendations on ways to increase instructional effectiveness 
which, if implemented, will lead to an increase in student 
achievement. Such an evaluation may well be considered worth the 
dollar cost. Several issues arise here. The first is cost 
effectiveness:' the dollar cost of the evaluation relative to the 
resulting increase in instructional effectiveness (and hence in 
student achievement } • The second issue relates to the 
economist's concern with indirect costs in the second sense of 
the terms opportunity costs. Granted that the benefits of an 
evaluation — in terms of increased student achievement — exceed 
the costs of the evaluation, it is nonetheless possible that 
alternative uses of the evaluation funds might have led to even 
greater benefits (i.e., higher student achievement). 

The third issue is more complicated: The recommendations of 
an evaluation are not always implemented, so one cannot always 
determine whether they produce benefits. Some authorities would 
take the position that, in such instances, one should consider 
the potential benefits that would have accrued, had the 
recommendations been implemented, and use those as a basis for 
judging the appropriateness of the costs. Other authorities 
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would maintain that; i£ the evaluation was not persuasive enough 
to convince decision-makers that its recommendations should be 
implemented, then the evaluat.ion has failed and cannot' be 
regarded as having produced benefits. The question comes down to 
this: Should one look at potential or at actual benefits? Is 
evaluation only a type of research, whose benefits are to be 
judged by the nature of the research findings? Or is evaluation 
a decision-oriented interactive process > whose benefits are to be 
judged not so much by the recommendations it makes but by the 
extent to which it beneficially informs the decision process? 
Obviously, the answers one gives to these questions will 
determine, in large part, just how one calculates the 
cost-benefit equation. 

Cost Implications of High-Utilization Factors 

The Evaluation Productivity Project has, over a period of 
years, been concerned with identifying those factors associated 
with instances of high evaluation utilization (see, for example, 
Alkin, Daillak, & White, 1979; Daillak, 1980; Stecher, Alkin, & 
Flesher, 1981). During the current fiscal year, we have 
completed a handbook for evaluation decision-makers that includes 
a factor framework (Alkin, Jacobson, Burry, Ruskus, White, & 
Kent, in press). 

One conclusion t<^ emerge from our years of research is that 
the way in which the evaluation is conducted affects the extent 
to which its findings are utilized. As mentioned in the previous 
section, the real issue in judging the appropriateness of an 
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evaluation's cost is the extent to which the benefits exceed the 
cost. Tf the findings of an evaluation are not utilized at any 
level, then it is questionable whether even the most minimal 
evaluation costs are justified. At the same time, one must 
recognize that the very aspects of the evaluation most likely to 
increase its use are also likely to involve high costs in terms 
of both dollars and also the time and energy of almost everyone 
involved- 

The results of our studies have not been surprising. We 
have found that the dedicated evaluator — the one who takes the 
time to understand the political complexities of the evaluation 
situation, to consider the needs and interests of potential 
users, and to involve them in the planning and conduct of the 
evaluation — stands a better chance of seeing the findings of 
the evaluation utilized. But at the same time, this approach 
incurs some heavy costs, at least in terms of the evaluator 's 
time and the time of program staff. Similarly, evaluation 
utilization increases when the potential „ user — the program 
administrator who commissioned the evaluation — is interested in 
the evaluation, is involved in the evaluation process, and is 
committed to integrating the evaluation findings into the 
decision process. Obviously, this degree of involvement on the 
part of potential users entails substantial indirect costs in the 
form of forgone opportunities to use their time in other, 
possibly more productive, ways. 

Let us look more closely at some of the factors just 
mentioned, starting with the political sensitivity of the 
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evaluator. The politically sensitive evaluator must be 
particularly attentive to the people within the organization, 
their place in the administrative structure, their special 
interests, and so forth. Such attentiveness may require a 
considerable outlay of the evaluator 's time in becoming familiar 
and dealing with the situation. Moreover, if sensitivity to, 
multiple constituencies adds to what must be known, there may be 
additional costs associated with extended data collection and 
data processing. In addition to these direct costs, there are 
indirect costs: for instance, the time which progpam . personnel 
spend interacting with the, evaluator, the time which clients or 
pupils spend taking tests. 

The evaluator 's credibility is another factor that affects 
utilization. Credibility depends in part upon the evaluator 's 
credentials (e.g., academic or professional degrees, prestige or 
reputation, institutional affiliations, experience). Obviously, 
a highly credent ialed evaluator costs more than a relatively 
uncredentialed evaluator: in consultant fees for external 
evaluators and in salaries for evaluators internal to the 
organization. Our research shows, however, that credibility is a 
function not only of the evaluator 's credentials at the outset of 
the evaluation but of the evaluator 's actions during the course 
of the evaluation: "As evaluators engage in their activities, 
they may come to be viewed as credible on a wider range of topics 
or credible to new audiences." (Alkin et al., 1979, p. 247). In 
short, an evaluator can build credibility, but this process is 
time-consuming and involves both direct and indirect costs. 
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As indicated earlier, the potential user is also a key 
figure' in evaluation utilization. "The more the potential user is 
concerned about the evaluation, iftvplved in its conduct, and 
interested in its results, the greater the likelihood that the 
findings of the evaluation Will be utilized. Obviously, this 
kind of usef'^nvolvement has cost implications, especially in 
terms of indirect costs. And the larger the number of potential 
users, the higher the costs. 

Not only evaluator and user characteristics but also obtain 
project characteristics have cost implications. One example is 
the particular requirements that the organization imposes upon 
the evaluation. To the extent that these contractual obligations 
or written requirements are not directly relevant to the central 
concerns of the evaluation or the interests of the potentia'l 
users, they may entail high costs — in terms of the time and 
research spent by the evaluator and the time spent by program 
personnel in interacting with the evaluator — and yet have no 
accompanying payoff in high utilization. 

The evaluation procedures, including design and methodology, 
obviously have cost implications. To the extent that these 
procedures are tailored to the particular needs of the program 
being evaluated and are applied rigorously, their costs may be 
high. These costs include outlays for materials, data 
processing, and so forth; the time and effort of the evaluation 
staff and program personnel; and client time spent in testing. 
Nonetheless, appropriate procedures are essential to high 
utilization, insofar as the way in which the evaluation is 
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conducted influences potential users' perceptions of the 
evaluator's credibility and of the quality of the evaluation. 

Finally, evaluation reporting, which is strongly related to 
utilisation, has cost implications. To prepare a report that can 
be easily' understooc? by all potential audiences and to 
disseminate it in a timely manner requires considerable effort on 
the part of the evaluator, and thus considerable cost. 
Similarly, follow-up procedures — designed to assure that users 
understand the report and its recommendations — call for extra 
time and effor^ and therefore carry extra costs. Our research 
shows that siii.pXy preparing and distributing a report of the 
evaluation is not enough, if high utilization is desired. 
Evaluators should regard evaluation as a process, not a product. 
They should view evaluation reporting as an almost-continuous set 
of activities designed to sensitize potential users to the 
information being developed, to prepare them for the findings 
that will emerge from the evaluation, and to encourage them to 
implement the recommendations. Without question, such activities 
entail high costs, both direct and indirect. Nonetheless, they 
are essential to attaining high levels of evaluation use. 

Summary 

t 

These reflections on the costs of evaluation represent a 
brief summary of the work we have completed in this area and of 
the implications that can be drawn from it. In synthesizing our 
reflections, we have drawn heavily on The Costs of Evaluation 
(Alkin & Solmon, 1983). Many authors contributed chapters to 



this volume, and some of their ideas are incorporated into this 
piece. Our perspectives on the costs associated with various 
utilization factors grew out of our previous effort in developing 
the utilization framework. Thus, the paper provides a synopsis 
of our work relative to the costs of evaluation, particularly 
evaluation which has a high potential for utilization. 
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